Answering questions with data

• We conduct a thought experiment to see whether our results could have occurred by chance

• Question: What would the world look like if we knew the truth?

  • Average treatment effect is \(0\)
  • Each individual effect is \(0\)
  • Sample mean is equal to \(X\)
  • …

Hypothesis testing

• Hypothesis test: (a) Assume some value other than what you observe is true, (b) determine what the data would look like in that world, (c) compare this to what you observe

Null and alternative hypothesis

• Null hypothesis: Some statement about the population parameters.

  • The “devil’s advocate” hypothesis \(\Rightarrow\) assumes what you seek to prove wrong
  • Ex: Biden’s approval is the same as election result
  • Ex: Treatment effect is zero for everyone
  • Denoted \(H_0\)

Practicum example

• Parameter: Average Treatment Effect (ATE) \(\mu_T − \mu_C\) of encouragement on support for government policies

  • \(\mu_T\): Average support for government policies if everyone received encouragement
  • \(\mu_T\): Average support for government policies if no one received encouragement

\(p\)-value

• \(p\)-value (based on a two-sided test): Probability of getting an (absolute) difference in means this big (or bigger) if the null hypothesis was true

  • Lower \(p\)-values \(\Rightarrow\) stronger evidence against the null
  • Intuition: How likely are we to observe what we observe if the null hypothesis is true?

How to calculate?

• Using CLT as we did in the practicum

  • \(SE(\bar{Y}_{\text{treated}} - \bar{Y}_{\text{untreated}}) = \sqrt{\frac{\sigma_{\text{treated}}^2}{n_{\text{treated}}} + \frac{\sigma_{\text{untreated}}^2}{n_{\text{untreated}}}}\)
  • See how many SEs away from null hypothesis is our observed difference-in-means and use normal distribution properties to calculate \(p\)-value

RI: Observed Practicum Data

RI: Practicum Data under Sharp Null

Permutation 1

Permutation 2

Permutation 3

Permutation 10

Permutation 100

Permutation 1000

Permutation 10000

Resulting \(p\)-values are similar!

Z <- c(1    , 1    , 1    , 0    , 0.    , 0.    , 0.   , 0.    , 1.   , 1)

Y <- c(0.859, 1.930, 0.875, 2.944, -1.015, -0.064, 1.624, -0.411, 1.048, -0.282)

dim_observed <- mean(Y[Z == 1]) - mean(Y[Z == 0])

se <- 
  sqrt(
    var(Y[Z == 1])/length(Y[Z == 1]) + 
      var(Y[Z == 0])/length(Y[Z == 0])
  )

c(estimate = dim_observed,
  std.error = 2 * pnorm(q = abs(dim_observed), mean = 0, sd = se, lower.tail = FALSE))

 estimate std.error 
0.2704000 0.7382426 

stats::lm(Y ~ Z) |> 
  summary() |> 
  broom::tidy() |> 
  dplyr::filter(term == "Z") |>
  dplyr::select(estimate, p.value)

# A tibble: 1 × 2
  estimate p.value
     <dbl>   <dbl>
1    0.270   0.747

dim_observed <- mean(Y[Z == 1]) - mean(Y[Z == 0])

dim_simulated <- 
  base::replicate(1000, {
    Z_simulated <- sample(Z)
    mean(Y[Z_simulated == 1]) - mean(Y[Z_simulated == 0])
  })

c(estimate = dim_observed,
  p.value = mean(abs(dim_simulated) >= abs(dim_observed)))

estimate  p.value 
  0.2704   0.7430 

Testing errors

• A \(p\)-value of \(0.05\) says that data this extreme would only happen in 5% of repeated samples if the null were true.

  • In other words our test results are not always correct!
  • 5% of the time we’ll reject the null when it is actually true.

College sports 🏈 and elections 🗳️

• Irrelevant events affect voters’ evaluations of government performance by Healy, Malhotra, and Mo (2010)

• Summary:

  • Previous research finds that natural disasters and other unpredictable events have an effect on incumbent politicians’ support
  • Authors argue that even events completely unrelated to politics can have an effect on incumbent’s support
  • Consider case of college footbal and basketball games
  • Mix of observational and survey experimental evidence
  • Several robustness excercises

Digging into argument

• Should we expect events unrelated to politics to have an effect on voting?

  • Do government investments and response matter?
  • Do psychological factros matter?
  • What else could matter?

Study 1: College Football and Elections

• What are the dependent and independent variables?

• How are those operationalized?

Controls

• What does it mean to control for the factors (covariates) in estimation? Why would we do that?

Robustness and Placebo tests

• What is robustness test?

Evidence

Study 2: Survey experiment

• What are the dependent and independent variables?

• How are those operationalized?

Evidence

Story of false-positives 😱

• College football, elections, and false-positive results in observational research by Fowler and Montagnes (2015)

How do they show this

• What is dependent and independent variables? How are they operationalized?

• What tests do they run? Why do they run the same regression initially?

How do they explain these results

• Fowler and Montagnes (2015) acknowledge that Healy, Malhotra, and Mo (2010) also ran robustness/placebo tests, BUT:

Broader implications for research design

• Think about all possible implications of theory and see if you can test them with your or other available data

  • This is especially important for observational studies

• Operationalization of variables is crucial!

• If you run many tests of the same hypothesis you need to adjust for multiple comparisons since you can come up with significant results just by chance